A synthesis process using a plurality of continuously photographed images is known to be effective as an image noise reduction (NR) process. Specifically, a pixel value in which noise is reduced is calculated by detecting a corresponding pixel serving as a photographing pixel region of the same subject from a plurality of continuously photographed images and performing, for example, a synthesis process of averaging pixel values of a plurality of corresponding pixels.
A technique of performing noise reduction by continuously photographed images is a process using three-dimensional (3D) data in which a plurality of images at different photographing times as well as a two-dimensional region of one image are included, that is, an image in a time axis direction is also added and thus referred to as a 3D noise reduction process, that is, three-dimensional noise reduction (3DNR).
FIG. 1 illustrates an exemplary configuration of a device that performs an image noise reduction process using the 3DNR.
A 3DNR processing unit 20 illustrated in FIG. 1 sequentially receives input images 11 serving as images photographed by a camera. For example, the input images 11 are continuously photographed images each of which includes noise as indicated by an input image sequence in FIG. 1(a).
The 3DNR processing unit 20 receives the input image 11 and an intermediate synthetic image 12 generated based on a previous input image, and calculates a synthetic ratio (α) of corresponding pixels of the two images through a synthetic ratio (α) calculating unit 21.
For example, in the case of a static region, averaging is performed in a state in which an addition ratio of pixel values of corresponding pixels of the input image 11 and the intermediate synthetic image 12 is set to 1:1. A ratio at which added noise is smallest in terms of a noise estimation value of each pixel may be caused as the addition ratio. In the case of a moving region, a synthetic ratio calculation in which a synthetic ratio is set such that the addition ratio of the pixel value of the input image 11 is increased is performed.
For example, it may be determined whether or not a corresponding region is a moving region by comparing a sum of absolute differences (SAD) of pixel values of pixel regions at corresponding positions of respective images with a noise level. In other words, when a differential value is sufficiently larger than a noise level of a target pixel, a corresponding subject can be regarded as a moving subject.
A process of generating an output image based on a plurality of continuously photographed images is used in a dynamic range-expanded image generation process using continuous-photographed images of different exposure conditions as well as the noise reduction process. The dynamic range-expanded image is an image in which high accuracy pixel values are set from a low brightness region to a high brightness region and referred to a wide dynamic range (WDR) image.
A process of generating a WDR image using continuous-photographed images of different exposure conditions is performed as a process of continuously photographing a small exposure amount image having a small exposure amount and a large exposure amount image having a large exposure amount, selectively using or combining (blending) effective pixel values included in the respective images, and setting pixel values of an output image. Through this process, it is possible to generate the WDR image in which effective pixel values are set from a low brightness region to a high brightness region.
Specifically, pixel values of an image having a small exposure amount (a small exposure amount image) are preferentially used for a saturation pixel region included in an image having a large exposure amount (a large exposure amount image), and pixel values of a large exposure amount image is preferentially used for a noise region of a small exposure amount image. By combining a plurality of images having different exposure amounts, it is possible to generate a dynamic range-expanded image in which high-accuracy effective pixel values are set from a low brightness region to a high brightness region. At this time, a method of changing an exposure amount may be implemented, for example, by a method of changing an exposure period of time, a method of changing an F value (diaphragm), or a method of inserting a neutral density filter having a different light transmittance each time of photographing. Further, an exposure amount may be changed by a combination of the above-mentioned methods.
FIG. 2 illustrates an exemplary configuration of a device that performs a WDR image generation process.
A WDR processing unit 40 illustrated in FIG. 2 sequentially receives input image 31 serving as an image photographed by a camera. For example, the input images 31 are continuous-photographed images of different exposure conditions as indicated in an input image sequence in FIG. 2(a).
FIG. 2(a) illustrates an image sequence of continuous-photographed images of exposure conditions R1 to R4 as an exemplary input image sequence.
The exposure amounts of the images have the following magnitude relation:
R1<R2<R3<R4
For example, images of four different types of exposure conditions are regarded as one WDR set, and one WDR image is generated by combining configuration images of the WDR set and output. When a moving image is photographed, the WDR sets of the exposure conditions R1 to R4 are consecutively acquired, and one WDR image is sequentially generated from each WDR set. A frame rate of an output image is lower than a frame rate of a photographed image, but the output image becomes a wide dynamic range image in which high-accuracy pixel values are set from a low brightness region to a high brightness region.
The WDR processing unit 40 illustrated in FIG. 2 receives the input image 31 and an intermediate synthetic image 32 generated based on a previous input image, and calculates a synthetic ratio (α) of corresponding pixels of the two images through a synthetic ratio (α) calculating unit 41.
As described above, when the WDR image is generated, pixel values of a small exposure amount image having a small exposure amount are preferentially used for a saturation pixel region included in a large exposure amount image having a large exposure amount, and pixel values of a large exposure amount image having a large exposure amount are preferentially used for a noise region in a small exposure amount image having a small exposure amount.
The synthetic ratio (α) calculating unit 41 detects a pixel region having an effective pixel value and a saturation region or an ineffective region having large noise in the input image 31 according to the exposure amount of the input image 31, and sets a synthetic ratio of the pixel value of the input image 31 to be higher than that of the pixel value of the intermediate synthetic image 32 for the effective pixel value region. On the other hand, for the ineffective region, the synthetic ratio of the pixel value of the input image 31 is set to be lower than that of pixel value of the intermediate synthetic image 32.
A synthesis processing unit 42 performs the process of synthesizing pixel values of corresponding pixels of the input image 31 and the intermediate synthetic image 32 according to the synthetic ratio (α) calculated by the synthetic ratio calculating unit 41, and generates a WDR output image 33.
The image generated by the synthesis processing unit 42 is used as a next intermediate synthetic image 32 until processing of one WDR set unit ends and then output as the WDR output image 33 when processing of one WDR set unit ends.
Both the noise reduction process described above with reference to FIG. 1 and the WDR image generation process described above with reference to FIG. 2 are necessary to acquire a high-quality output image, and a device that sequentially performs the two processes has been proposed.
As documents disclosing a configuration of performing a noise reduction process and a wide dynamic range process together through an image synthesis process, there are, for example, Patent Document 1 (Japanese Patent Application Laid-Open No. 2010-041200), Patent Document 2 (Japanese Patent Application Laid-Open No. 2008-160881), Patent Document 3 (Japanese Patent Application Laid-Open No. 2007-049227), and Patent Document 4 (Japanese Patent Application Laid-Open No. 2006-311240).
An exemplary configuration of a device that performs two processes, the noise reduction process described above with reference to FIG. 1 and the WDR image generation process described above with reference to FIG. 2, will be described with reference to FIGS. 3 and 4.
FIG. 3 illustrates an exemplary configuration of an image processing device in which a 3DNR processing unit 53 performing noise reduction functions as a preceding processing unit, and a WDR processing unit 57 performing wide dynamic range image generation functions as a subsequent processing unit.
On the other hand, FIG. 4 illustrates an exemplary configuration of an image processing device in which a WDR processing unit 65 performing wide dynamic range image generation functions as a preceding processing unit, and a 3DNR processing unit 67 performing noise reduction functions as a subsequent processing unit.
In both configurations, input images 51 and 61 are a repeated input of continuous-photographed images set to different exposure conditions, that is, a repeated input of a WDR set including continuous-photographed images set to different exposure conditions illustrated in FIG. 2(a).
Here, a problem lies in that the 3DNR processing unit performing the noise reduction has to perform a synthesis using a plurality of images of the same exposure condition, and the WDR processing unit performing the wide dynamic range image generation has to perform a synthesis process of synthesizing a plurality of images of different exposure conditions.
In the configuration illustrated in FIG. 3, the 3DNR processing unit 53 performing the noise reduction functions as the preceding processing unit, and includes frame memories a to d that store images having four different exposure amounts R1 to R4 illustrated in FIG. 2(a), respectively.
The frame memories have a double buffer configuration including memories 54a to 54d and memories 56p to 56s, respectively.
In other words, each of a set of the memory 54a and the memory 56p, a set of the memory 54b and the memory 56q, a set of the memory 54c of the memory 56r, and a set of the memory 54d and the memory 56s has a double buffer configuration, and performs reading and writing alternately.
For example, the image of the exposure condition R1 is written in the frame memory a 54a, and images of an immediately previous WDR set are read from the frame memory p 56p. 
Thereafter, the image of the exposure condition R1 is written in the frame memory p 56p, and images of an immediately previous WDR set are read from the frame memory a Ma.
The writing and the reading are alternately performed as described above.
Similarly, the image of the exposure condition R2 uses the frame memory b 54b and the frame memory q 56q. 
The image of the exposure condition R3 uses the frame memory c 54c and the frame memory r 56r. 
The image of the exposure condition R4 uses the frame memory d 54d and the frame memory s 56s. 
A setting is performed as described above.
Through the Double buffer configuration, a smooth process can be performed without adjusting a write timing and a read timing.
The 3DNR processing unit 53 serving as the preceding processing unit selectively reads a synthetic image based on a previous input image having the same exposure condition as an exposure condition of the input image 51 from the frame memory 56, and performs an image synthesis for noise reduction.
Further, the WDR processing unit 57 serving as the subsequent processing unit reads a set (WDR set) of noise-reduced images having different exposure conditions from the frame memories p to q 56p to 56q, performs a synthesis process in which effective pixel regions of each image are preferentially reflected, performs an image wide dynamic range process, and generates one output image.
Through a series of processes described above, an output image that has been subjected to the noise reduction process and the wide dynamic range process, that is, an output image 58 illustrated in FIG. 3 is generated and output.
On the other hand, in the configuration illustrated in FIG. 4, an input image 61 is alternately written in frame memories a to d 63a to 63d and frame memories p to s 64p to 64s according to each exposure condition. The four written images correspond to one WDR set described above with reference to FIG. 2(a).
Further, each of a set of the memory 63a and the memory 64p, a set of the memory 63b and the memory 64q, a set of the memory 63c and the memory 64r, and a set of the memory 63d and the memory 64s has a double buffer configuration, and performs reading and writing alternately.
The images written in the memory are read by the WDR processing unit 65, and one WDR image is generated through the wide dynamic range process using the images.
One WDR image generated in a WDR set unit including the images of a plurality of different exposure conditions by the WDR processing unit 65 is output to the 3DNR processing unit 67 serving as the subsequent processing unit.
The 3DNR processing unit 67 acquires the WDR image input from the WDR processing unit 65 and a previously input WDR image input from a frame memory z 68, performs a synthesis process of synthesizing the two images, and generates and outputs a new noise-reduced image. The output image is rewritten in the frame memory z 68 and output as an output image 69 illustrated in FIG. 4. At this time, the frame memory z may have a double buffer configuration using two frame memories.
In both of the configurations of FIGS. 3 and 4, there is a problem in that an operation circuit portion is increased in size since the synthesis process is independently performed, and a memory capacity necessary to temporarily store a plurality of images is increased.
In the configurations illustrated in FIGS. 3 and 4, the image synthesis process is performed without considering an alignment between images, but when an alignment is considered, a processing configuration is further complicated.
An exemplary configuration including a motion-compensated image generating unit that performs an alignment process of images of a synthesis target is illustrated in FIGS. 5 and 6.
Similarly to the configuration of FIG. 3, FIG. 5 illustrates an exemplary configuration of an image processing device in which a 3DNR processing unit 53 performing noise reduction functions as a preceding processing unit, and a WDR processing unit 57 performing wide dynamic range image generation functions as a subsequent processing unit.
On the other hand, similarly to the configuration of FIG. 4, FIG. 6 illustrates an exemplary configuration of an image processing device in which a WDR processing unit 65 performing wide dynamic range image generation functions as a preceding processing unit, and a 3DNR processing unit 67 performing noise reduction functions as a subsequent processing unit.
In both configurations, input images 51 and 61 are a repeated input of continuous-photographed images set to different exposure conditions, that is, a repeated input of a WDR set including continuous-photographed images set to different exposure conditions illustrated in FIG. 2(a).
The configuration of FIG. 5 differs from the configuration of FIG. 3 in that a motion-compensated image generating unit 71 and motion-compensated image generating units 72p to 72r are added.
The motion-compensated image generating unit performs an alignment between images serving as a synthesis target.
In an image alignment process, first, an image serving as a motion matching reference is decided. A motion estimation (ME) process of estimating a motion of an image serving as an alignment target in view of the reference image is performed, and then a motion compensation (MC) process of performing an alignment of an image according to an estimated motion is performed.
The motion-compensated image generating unit 71 and the motion-compensated image generating units 72p to 72r perform the process (MEMC) and perform an alignment between images serving as a synthesis target.
In FIG. 5, the image of the exposure condition R4 is used as the reference image for the alignment, but the reference image is not limited to this example, and any of the exposure conditions R1 to 4 may be used.
The configuration illustrated in FIG. 6 is a configuration in which motion-compensated image generating units 81p to 81r and a motion-compensated image generating unit 82 are added to the configuration illustrated in FIG. 4.
The motion-compensated image generating units 81p to 81r and a motion-compensated image generating unit 82 also perform motion estimation (ME) and motion compensation (MC) between images, and perform an alignment between images serving as a synthesis target.
In the configuration in which an alignment between synthetic images is considered as illustrated in FIGS. 5 and 6, there is a problem in that compared to the configurations illustrated in FIGS. 3 and 4, a configuration is more complicated, and a size of hardware performing the above processes is increased.
As documents disclosing a configuration of performing a noise reduction process and a wide dynamic range process together through an image synthesis process, there are, for example, Patent Document 1 (Japanese Patent Application Laid-Open No. 2010-041200), Patent Document 2 (Japanese Patent Application Laid-Open No. 2008-160881), Patent Document 3 (Japanese Patent Application Laid-Open No. 2007-049227), and Patent Document 4 (Japanese Patent Application Laid-Open No. 2006-311240).